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Introduction 


State funding methods for public colleges and universities has varied over time, with many of the 
changes in approach reflective of evolving priorities and goals for higher education. Today, there is a 
mix of funding approaches used by states, again reflective of the particular circumstances within each 
state (Hearn, 2015). One popular method used by states to fund postsecondary institutions is by the 
number of semester credit hours of instruction that students enroll in. These enrollment-based models 
are generally viewed as fair, although flagship campuses with limited ability to grow are not always 
enamored. States expend considerable effort to develop and achieve buy-in from colleges and 
universities for such models, and many are calibrated with actual accounting data on historic delivery 
costs. However, because enrollment-based models tie funding solely to service volume, they provide 
little incentive for quality and may provide disincentives for timely student completion. 
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Recent years have seen rapid 


increases in both the need and 
demand for higher education, 
just as the economic downturn 
has placed increasing pressure 
on state higher education 
budgets. These trends and 
others have converged to 
inspire state policymakers, the 
coordinating and governing 
boards for higher education, 
and other stakeholders to 
consider ways to better align 
institutional priorities and 
activities with state goals, 
create incentives for quality, 
and more efficiently prioritize 
dwindling state resources for 


Recent years have seen rapid increases in both the 
need and demand for higher education, just as the 
economic downturn has placed increasing pressure 
on state higher education budgets. These trends 
and others have converged to inspire state policy- 
makers, the coordinating and governing boards 
for higher education, and other stakeholders to 
consider ways to better align institutional priorities 
and activities with state goals, create incentives for 
quality, and more etfticiently prioritize dwindling 
state resources for higher education. 
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higher education. There is increasing interest in considering new outcomes-based funding! models 


for higher education that explicitly tie state funding to well-defined performance metrics. Several 


states are developing or implementing outcomes-based funding models. 


Importantly, one criticism of the emerging outcomes-based funding models is that they tend to focus 


primarily upon productivity—specifically increasing the number of students who receive degrees and 
meet other academic milestones—without explicitly focusing on quality (Miao, 2012; Dougherty, 2012). 


'Outcomes-based funding models are an evolved form of “performance funding,” which refers to a broad set of policies linking allocation of resources to 
accomplishment of certain desired objectives. Historically, postsecondary performance funding models were often add-ons or bonuses to base institutional 
allocations that institutions earned for meeting various goals or benchmarks. Additionally, many of these earlier models included measures focused more on 
inputs or processes than student progression and outcomes and were not intended to drive increased student completion. Today's outcomes-based funding 
models similarly seek to create incentives for and reward progress toward a set of stated goals, and have a direct link to the states higher education 
attainment needs and place primary empbasis on student completion, though they often include measures beyond student progression and completion. 


Advanced outcomes-based funding models also determine bow a significant portion of the states general budget allocation to institutions is determined. 
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One criticism of the emerging 
Indeed, there is much concern that efforts to align funding outcomes based fundi ng 
with student outcomes may lead institutions to shirk 


on quality. By and large, these are valid and important models is that they tend to focus 


concerns, as few outcomes-based funding models have primarily upon prod uctivity... 
been designed in ways that more directly address issues . 
j Indeed, there is much concern 
of quality. 
that efforts to align funding with 
Meanwhile, just as outcomes-based funding models g g 
have gained popularity across states, the research student outcomes may lead 


community has turned towards the challenging task institutions to shirk on quality. 


of defining and measuring college quality. The hope 
is that rigorously defined quality metrics can be used 
to monitor quality as states implement outcomes-based funding models to ensure that efforts to 
improve student progression and completion do not come at the expense of quality. Eventually, 

if stakeholders can agree upon a set of quality metrics, they may be explicitly incorporated into 
outcomes-based funding models so that those models create incentives for both student progression 
and completion and quality. 


Unfortunately, we are a long way away from having a set of rigorous quality metrics that institutions 
and other stakeholders can agree upon. Defining and measuring college quality is a contentious issue, 
and the debate about how to do so is far from settled. What is quality when institutions produce 
outputs ranging from student learning and labor-market success to research? How do you measure 
student learning when students specialize in a major field? How do you tie quality metrics to funding 
in a way that recognizes the diverse missions of institutions? 


While these questions remain unanswered, states continue to move forward with outcomes- 
based funding models. Policymakers need useful information to guide these important decisions. 
Unfortunately, existing evidence comes from a myriad of disconnected and undigested sources such as: 


1) Academic and policy literature defining college quality; 


2) Experience with and assessments of a wide range of metrics designed to measure components 
of college quality; 


3) States that have developed and implemented an outcomes-based funding model. 


This paper distills these existing sources into the most important lessons for policymakers 
considering outcomes-based funding models and wanting to ensure that those models improve student 
progression and completion while also maintaining or enhancing quality. We begin by discussing 
the concept of quality in higher education, paying particular attention to the multiple and often 
differentiated objectives of colleges and universities, as well as possible indicators of quality factors 
that could be measured in different ways. In doing so, we are careful to make a clear distinction 
between quality on the one hand and productivity on the other. Next, we describe criteria by which 
to evaluate quality metrics. We then describe quality metrics that researchers and policymakers have 
used or proposed, and we evaluate those metrics by the criteria we describe. We also describe the 
outcomes-based funding models that states have used for tying funding to outcomes, and offer in- 
sights about how current and future quality metrics might be incorporated into those models or used 
to ensure outcomes-based funding models do not lead to reduced quality. We close with overarching 
recommendations based on the existing evidence. 
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Important Considerations for Defining Quality in Higher Education 


Measuring college quality presupposes a rigorous and well-defined notion of quality in higher 
education. Unfortunately, the research and policy community is far from consensus on this. Below 
are several of the greatest challenges to developing a rigorous notion of quality in higher education. 


Colleges and universities draw upon ; ; 
a range of inputs and processes to Measuring college quality PreESUPPOSes A 


produce a diverse array of important rigorous and well-defined notion of quality 
outputs and outcomes. Colleges and a aah d fi Untort rely Hy 
universities are complex institutions that Te ee a 

endeavor to produce many different outputs research and policy community iS far from 


and achieve a number of different goals. eencensie Getic. 


They seek to promote: 


1) Student success. First and foremost, 
colleges and universities endeavor to improve the outcomes and life experiences of their students. 
Students draw on the knowledge and skills they gain in college to excel in their careers and be 
knowledgeable, engaged and productive members of society. 


2) Student access and diversity. Colleges and universities seek to provide opportunities to 
non-traditional students, including minority, economically disadvantaged and adult students. 
Colleges and universities increasingly recognize the value that diverse students bring to campus 
life. If national goals for attainment are to be met, policymakers and higher education leaders 
increasingly must place a priority on eliminating inequitable educational outcomes among 
racial and ethnic minorities and other underserved students. 


3) Meeting workforce needs. Colleges and universities seek to connect their students to employers 
who can benefit from their knowledge and skills. Many colleges and universities work with 
large employers and other stakeholders to develop programs specifically tailored to employer 
needs. Others equip students with general skills for use in a broad array of careers. 


4) Research and innovation. Many colleges and universities produce scholarly and applied research 
that benefits the academic community and society. 


Colleges and universities have differing missions. Colleges and universities have wide-ranging 
missions with differing emphases on different outputs and outcomes. Institutions also vary considerably 
in the processes they employ to achieve their missions. For example, one institution may generate 
millions of dollars of externally funded research, while another may produce no research at all. Similarly, 
acommunity college may develop degree programs specifically targeted at meeting local workforce 
needs, while a liberal arts college focuses on equipping its students with general knowledge and skills. 
Nevertheless, each institution may perform equally well when measured 
by its mission and goals. 


Colleges and universities have different students and resources. Colleges and universities 
vary greatly in the students they serve and the resources they have to produce the outputs they 
prioritize. State flagship universities, for example, tend to have students with more pre-existing 
knowledge and skills as well as a larger endowment that can be used to hire high-quality faculty and 
develop programs that contribute to the success of their students (Cunha and Miller 2014; Hoekstra 
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Colleges and universities have 


2009). It is thus no surprise that earnings and other wide-ranging missions with 
outcomes of graduates of flagships are better on : ; : 
average than those of students attending regional ditteri NG emphases on different 


bachelor's-degree-granting institutions. A portion outputs and outcomes. 
of the resulting difference in outcomes between ae 
these institutions is attributable to the types of Nevertheless, each institution 

students attending them and the resources available may perform eq ually well when 
to them (Cunha and Miller 2014; Hoekstra 2009; 


Schneider 2014). measured by its mission and goals. 


Institution priorities often differ from the state's 
objectives. An institution's mission and goals may conflict with the best interests of the higher 
education system or with the goals of policymakers. For example, a regional PhD-granting institution 
may desire to expand its mission and greatly increase POVERUTITESELTTTTTT TILT E TITTLE ITT PTET T errr reer 


its research production. Similarly, a community college A key goal for outcomes-based 
may desire to offer bachelor's degrees in applied fields. 


While both may be worthy objectives for an ambitious tunding models should be to 
institution, such goals may compete with those of other promote alignment between 
institutions that are already addressing these needs. A the state's objectives and those 
key goal for outcomes-based funding models should be ee 
to promote alignment between the state's objectives and of institutions. 
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Quality must be clearly distinguished from productivity and efficiency. Productivity and 
efficiency are worthy goals of states and institutions. However, critics of outcomes-based funding 
models have argued that focusing on productivity and efficiency could cause institutions to reduce 
the quality of education they offer. The goal of this paper is to address this criticism, so any quality 
metric must focus on institutional performance relative to other goals. We describe some key goals 
and metrics below. 


Criteria for Quality Metrics 
A good metric of quality should: 


Flow directly from the needs and goals of students, the state, or the mission and goals of 
an institution, or a higher-education system. As noted above, institution missions may include 
promoting student success, promoting student access and diversity, meeting workforce needs, and 
research. To a large extent, student and state needs and goals overlap with these primary focus areas, 
but the relative weight each group places on those areas can differ considerably across groups. For 
example, research institutions tend to place more weight on research than students or the state, 
particularly given the federal role in funding research in the United States. While, in general, a 
quality metric should be informed by the needs and goals of other stakeholders, a key goal of 
outcomes-based funding is to better align the incentives of institutions with those of the state. 

We thus focus on quality metrics that can help achieve this goal. 
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A good quality metric should focus 


Be focused on achieving outputs or on producing outputs and outcomes. 
outcomes. A good quality metric should focus 
on producing outputs and outcomes. Input Many of the most popular college 


and process metrics are also relevant when they ranking systems, such as that by 
can be shown to produce desired outputs and 

U.S. News & World Report, use 
outcomes. Many of the most popular college 
ranking systems, such as that by U.S. News « inputs such as studenHaculty ratios and 
World Report, use inputs such as student-faculty 


ratios and selectivity that are not clearly linked selectivity that are not clearly linked to 


to important outputs and outcomes. These important outputs and outcomes. 
tankines reward instittitionsfor the students —-—-» sssetacesaasadsvecendesesastag see seccteusanssesecnacanaanebesaesecasscaaess 
they serve, rather than the value they add. 


Adjust outcomes and outputs for A foi . finden . 
inputs. A good quality metric should adjust See eT ayers is avi 


outcomes and outputs for the inputs that should account for differences in the 
Il d universities have t d aT . : 
CONCBES ann amiversies Nae #0 pros characteristics of students (i.e., academic 
them. For example, a fair comparison of 
student earnings should account for differ- reparation at time of entry) as well 
ences in the characteristics of students as other pre-existing reSOUrCes ACTOSs 
(i.e, academic preparation at time ofentry) 
as well as other pre-existing resources across institutions. 
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Be straightforward and understandable. Institutions and policymakers must have a clear under- 
standing of the metrics to which they are held accountable and of the particular way in which they 
contribute to funding. This allows them to institute effective policies to improve their performance. 
Complex statistical methods to adjust outcomes and outputs for inputs may be attractive for precision 
but difficult for colleges, universities, and policymakers to understand. 


Be practical and cost-effective to implement. Wherever possible, the metric should minimize the 
need for additional data collection by drawing on existing data resources such as state administrative 
data systems and federal data such as the Integrated Postsecondary Education Data System (IPEDS). 


Provide short- to medium-term 


feedback. In order to induce meaningful In order to induce meaningful change, 


change, a quality metric should provide gq quality metric should provide short to 


short- to medium-term feedback to colleges ; 
nea medium-term feedback to colleges and 

and universities so that they can receive a 

positive and timely return on investments universities so that they can receive a 

in processes and activities intended to 


positive and timely return on investments. 
improve their performance on the metric. 


Be difficult to game. Experience in other sectors suggests that policies that create incentives 
for performance on a set of metrics may elicit unintended responses by institutions interested in 
artificially increasing their performance on those metrics. For example, in the K-12 context, there is 
evidence of increased suspensions of students likely to perform poorly on state assessments on days 
when tests are administered (Cullen and Reback, 2006). 
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There can be tension between some of the criteria listed above. Some metrics may do well on some 
criteria and poorly on others. For example, while all colleges and universities presumably endeavor 
to improve earnings outcomes of their graduates, it takes five to six years post-enrollment to observe 
earnings for a meaningful share of an entering cohort of students. Thus, while the input-adjusted 
earnings of graduates metric does well on many of the criteria listed above, it may be limited in utility, 
as it does not provide meaningful short-term feedback to colleges and universities. 


Additional Criteria for Embedding Quality Metrics into 
Outcomes-Based Funding Models 


An outcomes-based funding model requires both a set of good quality metrics and a funding 


methodology that encourages, rather than discourages, quality. A good outcomes-based funding 
model should: 


Promote alignment between state priorities and the mission and goals of institutions. As 
noted above, institutions often have competing interests with students and the state. An overarching 
goal of outcomes-based funding models should be to use state funding to make institutions more 
accountable to the mission and goals of students and the state. As such, the quality metrics embedded 
in the outcomes-based funding model should be oriented toward the state's perspective. 


Include an appropriate mix of input, process, and outcome metrics. Specifically: 


Place significant weight upon appropriate output and outcome metrics. Outcomes-based 
funding models should hold institutions accountable for achieving results and pursuing activities 
aligned with state goals, and hence should place significant weight upon output and outcome 
metrics. As noted above, outcomes and outputs should account for differences in student charac- 
teristics and other relevant inputs, so as to make fair comparisons across institutions. 


Include input metrics to create incentives for success with underserved students. In 
some cases, the state may also wish to provide incentives for institutions to effectively serve 
certain groups of students, such as racial and ethnic minorities, Pell Grant-eligible and / or adult 
students. In such cases, outcomes-based funding models may include input metrics, such as the 
number of full-time-equivalent students from each of these groups. Another approach used by 
several states currently implementing outcomes-based funding models is to put a premium (or 
extra “weight") for successfully serving underrepresented students. 


Aim for process metrics that create incentives for research-based activities that lead to 
important outputs and outcomes. Finally, while the ultimate goal of outcomes-based funding 
models should be to improve institution outcomes, these institutions may not know what activities 
to engage in to improve their performance on specific outcome and output metrics. Wherever 
possible, outcomes-based funding models can include significant weight upon process metrics that 
have been shown to improve performance on relevant outcome and output metrics. Indeed, current 
accountability systems in healthcare and, increasingly, in K-12 education include significant weight 
upon process metrics that have been shown to improve relevant outcomes (Williams et. al., 2005). 
Unfortunately, the research base in higher education currently lacks sufficient evidence to warrant 
inclusion of many process metrics, but over time, as research emerges, more weight can be placed 
upon relevant process metrics. 
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Account for differences in missions of colleges and universities. A successful outcomes- 
based funding model should recognize the varying missions of colleges and universities and encourage 
appropriate differentiation that leads to system-wide success. 


Ensure that outcomes-based funding is sufficient to 
gain attention but not too large to create instability. To The outcomes-based fu nding 
ensure that institutions have sufficient incentives to develop 


- system must have adequate 
and implement effective policies to support improvement, 


the outcomes-based funding system must have adequate tunding. At the same time, 
funding. At the same time, the funding tied to outcomes the fu nding tied to outcomes 
should be sufficiently stable to ensure that institutions can a 
adequately forecast funding streams and plan effectively. should be sutticiently stable 
To support effective change, studies recommend levels of | to ensure that institutions can 
outcomes-based funding ranging from 5 percent to 25 ; 
percent of overall state higher education funding (Miao, adequately forecast funding 
2013, HCM Strategists, 2013; Complete College America, streams and plan effectively, 
2012; Dougherty, Natow and Vega, 2010; National ———ceeeececececeeeeeeeeeereeeeeeeeeseeeseeeeeeeeeeeeeeeeeeeee 


Conference of State Legislatures, 2012). 


Achieve buy-in of colleges and universities and other stakeholders. A successful outcomes- 
based funding model must have the buy-in and support of all stakeholders to be successful and 
sustainable. Successful implementation requires close collaboration at all stages of design and 
implementation among the state government and institution administration and faculty (Kadlec 
and Shelton, 2015). 


Evaluating Current Metrics 


Researchers and policymakers have developed an array of metrics to measure contributors to 
college quality. Some of these metrics have been tied to state higher education funding. Below we 
describe the advantages and disadvantages of some of the most common college quality metrics in 
light of the criteria developed above. Given that a primary goal of outcomes-based funding models 
is to hold public institutions accountable for meeting state goals, we focus upon metrics that are 
focused on the state perspective. In particular, while research is an important activity of many colleges 
and universities, we do not focus on research metrics, given that the federal government plays the 
largest role in the research activities of colleges and universities. Instead, we focus upon student 
success, access and diversity, and meeting state and local workforce needs, as these are all areas that 
are closely aligned with state priorities. 


Student Success 


Student success metrics measure an institution's direct contribution to the outcomes and life 
experiences of students and alumni. These are direct measures of student outcomes, and measures of 
student learning engagement and satisfaction. 


Direct measures of student learning are attractive in theory and can be adjusted for 
student characteristics. In the short term, they show promise as a tool to monitor quality as states 
implement outcomes-based funding models. Student learning is a primary output of institutions, 
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which students draw upon to develop their careers and become Dinseh measures Sadan 
active and engaged members of society. Student learning is 

particularly important as critics of outcomes-based funding lea rning are attractive in 
have argued that efforts to improve productivity will lead to theory and can be adjusted 
watered down instruction and reduced learning. Despite the _ 
importance of student learning, developing a set of practical for student characteristics. 
metrics that accurately and fairly measure student learning In the short term, they show 
while also balancing the criteria for good college-quality . F 
metrics for outcomes-based funding is extremely challenging. Pompe Oso tool to monitor 
In practice, only Missouri has successfully implemented funding quality as states implement 
model that includes direct measures of student learning 
(National Governors Association, 2013).2 Nevertheless, these 


metrics show promise as a tool to monitor quality as states models. 


outcomes-based funding 


implement outcomes-based funding models, i eccccnneeneeseeetseseteeseteeeeeceeeeeeeeeeeneeeeees 


Measuring student learning in higher education is particularly complicated for two reasons. First, 
students customize their education by choosing a college major and the particular courses that lead 
to a credential or degree. Thus, the knowledge and skills that colleges and universities intend to 
impart on their students varies across students. Within the field of higher education, efforts such 
as the Degree Qualifications Profile have attempted to suggest common learning outcomes across 
such dimensions as the acquisition and application of: specialized knowledge, broad and integrative 
knowledge, intellectual skills, individual and collaborative learning, and civic and global learning. 


Second, colleges and universities develop two types of knowledge and skills: specific knowledge 
and skills, such as when a computer science major learns how to program in C++, and general 
knowledge and skills, such as the problem-solving skills developed across the entire curriculum. 
College majors vary in their degree of emphasis on general and specific skills, with engineering 
and nursing degrees weighted toward specific knowledge and liberal arts degrees weighted towards 
general knowledge. 


There are a number of assessments to measure general and specific skills, and many of these are 
well-validated.? Many of these tests are vertically scaled, allowing for the computation of value- 
added scores that help address differences in student characteristics across colleges and universities. 
Yet there are several reasons why direct assessment of student learning is unlikely to see wide use in 
outcomes-based funding models in the near future: 


1) It is difficult to interpret and apply these measures in a fair and transparent way that can 
achieve the buy-in of all colleges and universities. For example, how should we compare gains 
in general skills among engineering students to similar gains among English majors? 


2) Colleges and universities vary considerably in the extent to which the content covered in 
their degree plans aligns with the content covered in existing assessments. Given colleges 


2The Missouri funding model includes an indicator for scores on one of three types of assessments: general education assessments, major-specific assessments 


or licensure exams. Institutions must choose one type of assessment to include in their formula funding model (Coordinating Board for Higher Education). 


3These include the Collegiate Learning Assessment (CLA), ACTS Collegiate Assessment of Academic Proficiency (CAAP), and ETS's Measure of 
Academic Proficiency and Progress (MAPP). There are also existing assessments for specific skills in some but not all college majors. These include ETS 
Major Specific Tests, the GRE subject tests, and licensure exams. 
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and universities’ strong opposition to policies mandating curricula, it is likely to be difficult to 


achieve buy-in for a particular assessment. 


3) Institutions and other stakeholders have expressed concerns that directly creating incentives 


for performance on particular assessments will lead institutions to tailor their content towards 


the assessment and “teach to the test.” Indeed, there is evidence that teachers in K-12 schools 
have shifted their focus toward content covered in state tests (Reback, 2008). 


4) Wide implementation of assessments could be extremely costly, both in terms of test 


development and administration.* 


One specific type of student learning metric that has received significant attention is student 


performance on required certification and/or licensure exams, such as the NCLEX for nursing and the 


Fundamentals of Engineering, Professional Engineer and Structural Engineer exams for engineer- 


ing majors. These exams provide a useful and systematic source of data on the knowledge and skills 


of graduates in specific fields that require 
licensure. Because all graduates in fields that 
require licensure must pass these assessments 
in order to obtain relevant jobs within their 
fields, institutions place great emphasis on 
aligning curricula with the content covered 
in the assessments. Generally, there is broad 
consensus that the content covered in the 
assessments represents what reasonable 
graduates should know, so there is little 
concern over “teaching to the test.” Where 
student characteristic data are available at 
the program level, licensure test scores can 
be adjusted for student characteristics. 

When they can be adjusted for program-level 
entrance exams scores, they are similar to 
value-added metrics. The key limitation of 
these metrics is that they are only available at 
the program level, and only for programs that 
require licensure. 


One specitic type of student learning 
metric that has received significant 
attention is student performance on 
required certification and/or licensure 
exams. These exams provide a useful 
and systematic source of data on the 
knowledge and skills of graduates in 
specitic fields. The key limitation of 
these metrics is that they are only 
available at the program level, and 
only for programs that require licensure. 


Despite the challenges noted above, student-learning metrics deserve continued research. Because 


critics of outcomes-based funding have argued that increased attainment could come at the expense 


of student learning, it is important to consider how student learning can be measured, monitored 


and given incentives within the context of outcomes-based funding. Given the degree of alignment 


between program curricula and assessment content, student scores and pass rates on licensure exams 


show promise for direct inclusion within outcomes-based funding models. Broader student learning 


metrics could also be attractive as a monitoring tool for states that are implementing outcomes-based 


4However, the movement of colleges and universities toward competency-based learning as a way to plan, organize, deliver, and support education, 


which often uses analytics software to track the acquisition and demonstration of discrete competencies, offers significant potential in this area. 
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funding. For example, states could administer common assessments like the Collegiate Learning 
Assessment, Collegiate Assessment of Academic Proficiency, Measure of Academic Progress and 
Proficiency, or ETS Proficiency Profile subject tests to a random set of students from institutions 
across the state annually. If assessment scores decline as outcomes-based funding is implemented, 
this could be sign that institutions are shirking on quality and should redouble their efforts to 
maintain or improve quality. 


Direct measures of student outcomes are practical and cost effective to implement, but 
they should account for student characteristics and other available resources. Ultimately, 
colleges and universities endeavor to improve the life outcomes of their students. Data on student 
outcomes such as persistence, degree completion, and earnings are widely collected and readily 
available in different formats, making the calculation of student-outcome metrics relatively practical 
and cost-effective. Most of these metrics POeeee OU eREO eer reer reer reer rrrrr rer rrr rrr errr errr errr rrr errr errr errr rrr rrr rrr rrr rrr) 


can also be adjusted for student 


Se eee Nea ios Nearly all outcomes-based funding models 


based funding models include student include student outcomes metrics, but only 


outcomes metrics, but only a handful of . : 
| a handful of states directly adjust them 

states directly adjust them for student 

inputs. However, many states adjust for student inputs. However, many states 

student outcomes metrics indirectly by 


adjust student outcomes metrics indirectly 
placing greater weight on outcomes 


achieved by “at-risk” students, particularly by placing greater weight on outcomes 


those who are racial and ethnic minorities achieved by “atrisk” students, pa rticularly 
and/or from low-income families. States 


sisi work ecussdeqe plemented those who are racial and ethnic minorities 


outcomes-based funding models that and/or from low-income families. 
include student outcomes and adjust 


them for student inputs either explicitly 
or implicitly. 


Student outcomes metrics fall into two broad categories. 


1) Input-adjusted measures of progression. Degree- or credential-seeking students enter 
college intending to complete their course of study. These metrics aim to capture an institution's 
ability to support students in meeting their academic goals. Common indicators include 
measures of persistence towards degree, credit accrual, successful transfers, meeting key 
milestones, credential completion, and acceptance rates at graduate programs. These metrics 
have short- to medium-term time lags. 


2) Input-adjusted measures of labor-market success. Most students enter college to gain 
knowledge and skills to apply in the job market and meet their career goals. These metrics aim 
to capture an institution's ability to support students toward meeting their career goals. 
Common indicators include measures of job placement, earnings, and student loan default rates. 
These metrics have long time lags, which may limit their attractiveness as a policy-making tool. 


Researchers have developed a range of methods to directly adjust student outcomes metrics 
for student characteristics. One approach uses regression models to adjust for multiple student 
characteristics simultaneously. The goal of these methods is to control for any observable student 
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characteristic that might also be related to student outcomes, so as to make comparisons as fair as 
possible. Unfortunately, such methods conflict with the need for metrics to be simple and under- 
standable. Hence, they have been difficult for policymakers to implement in practice. 


An alternative and simpler approach that adjusts for some but not all student characteristics is to 
produce a set of outcome metrics for students with different characteristics, such as those groups 
defined by Pell Grant eligibility or high school grades. Policymakers have had more success in 
implementing metrics such as these that balance the need to account for student characteristics 
with the need to be simple and understandable. 


One criticism of approaches that directly adjust student outcomes metrics for student character- 
istics is that they effectively set lower expectations for “at-risk” students that have historically low 
success rates. An alternative approach is to adjust for student characteristics indirectly, by applying 
greater weights for outcomes achieved by at-risk students. This approach has achieved wider stake- 
holder buy-in because it rewards institutions for success with at-risk students, as opposed to applying a 
“lower bar." This approach is also straightforward and relatively simple to implement. For these reasons, 
it has been incorporated into the outcomes-based funding models developed and implemented. 
These measures are important for encouraging colleges and universities to adopt interventions aimed 
at improving outcomes for racial and ethnic minorities and other at-risk students. 


Student outcomes metrics vary considerably in the extent to which colleges and universities could 
game them. For example, colleges and universities could increase their completion rates by raising 
admissions standards or reducing the rigor or grading standards of their courses. Adjusting graduate 
numbers for student characteristics may limit potential gains from raising admissions standards, but 
does little to change incentives to reduce rigor or grading standards. Similarly, colleges and universities 
could increase earnings metrics by raising admission standards or shifting towards programs that 
offer higher earnings for graduates. Again, adjusting earnings for student characteristics may limit 
potential gains from raising admissions standards. Policymakers could also limit the incentive to shift 
towards programs that offer higher earnings by controlling explicitly for student major or computing 
earnings metrics by college major or broad field. 


Measures of student engagement and satisfaction are related to student learning and 
experiences but more appropriate for internal use by colleges and universities. Colleges and 
universities also seek to provide positive academic and life experiences for their students. Validated 
student survey instruments, such as the National Survey of Student Engagement (NSSE), capture 
factors related to student engagement. Many colleges and universities participate in the NSSE and 
use detailed reports provided to them to improve student experiences in and engagement with their 
academic programs. While these data indicate student engagement and academic experiences, the 
NSSE can be impractical and costly to sseesescceeeeessssceseeeccreeensssseseseceseeensssseseseceseeeesssseseseeeseeeneeeons 


implement, and the data are not publicly Measures of student engagement and 
available. It is also difficult to achieve 


eb tel elas aisnteecncia satisfaction are related to student learning 


as they address process issues that most. qnd experiences but more appropriate for 


colleges and universities prefer to control. . . a 
internal use by colleges and universities. 
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Student Access and Diversity 


Student access and diversity metrics attempt to measure an institution's success at attracting and 
serving diverse students and providing access to non-traditional students including minorities and adult 
learners. This class of metrics is unique in that they measure inputs and processes rather than outputs 
and outcomes. Nevertheless, access and diversity is often a goal in and of itself, both for colleges and 
universities and the state. Common metrics include the percentage of students receiving or eligible 
to receive Pell Grants, the percentages who are racial or ethnic minorities, or the percentage who are 
adult learners. These metrics are simple, cost-effective to implement, provide short-term feedback to 
colleges and universities, and are difficult to game. 


Meeting Workforce Needs 


Colleges and universities are increasingly called upon by local and state leaders to address work- 
force needs. Indicators of meeting workforce needs can be considered process metrics, as they support 
a key mission of colleges and universities and a state goal to improve student labor-market outcomes. 
The primary metric here has been number of degrees or credentials produced in "high-demand" 
fields. States vary considerably in how they define “high-demand" fields, with some employing 
complicated statistical methods, and some drawing on discussions with employers and other stake- 
holders. Most of these approaches are relatively new, and there is little research evaluating the 
relative merits of particular approaches. 


Whatever approach a state employsto = \\\/hatever approach a state employs to 
arrive at a set of high-demand fields, it should 


achieve stakeholder buy-in and: 


update the set of priority fields. 1) Institute a process to regularly update 
The process should balance the need 


arrive at a set of high demand fields, it 


the set of priority fields; and 


to give colleges and universities 
sufficient time to expand production — 2) Reward increased production rather 
in high-needs fields with the need 


for those fields to represent true than current production. 


workforce needs. 


2) Reward increased production rather than current production. Rewarding current production 
in high-needs fields would only increase funding for colleges and universities that already have 
high production in those fields rather than spur any colleges and universities to increase 
production in them. 


While meeting workforce needs is a laudable goal for colleges and universities, and one that 
states generally want to encourage, existing metrics are lacking in many ways. They are expensive 
to implement and it is unclear whether they accurately identify workforce needs. Further work is 
needed to design methods for identifying workforce needs. 
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Ensuring Quality 


While the primary focus of outcomes-based funding models has been upon improving student 
progression and completion, there are several ways in which these models have implicitly or explic- 
itly addressed quality: 


eer ere ee reer ere rrr rrr rere rrr rrr rrr rere rrr rr rrr rrr rer rere rrr reer rere 


Many past and current outcomes-based 
models are designed so as to implicitly In most cases, states have weighted 


include input-adjusted graduates. The degrees awarded to atrisk students 
primary focus of most current outcomes-based 


funding models has been overwhelmingly to (such as Pell Granteligible students, 


shift towards funding institutions on indicators qdylt learners or minority students) more 
of total degrees granted as opposed to enroll- . 
ment. However, in most cases, states have heavily. In such CoS the model does 
weighted degrees awarded to at-risk students account for student characteristics, and 
(such as Pell Grant-eligible students, adult 

learners or minority students) more heavily. 
In such cases, the model does account for on inputadjusted graduates, which is a 


student characteristics, and the net result is quality metric. 


the net result is similar to funding based 


similar to funding based on input-adjusted 


graduates, which is a quality metric. 


Many past and current outcomes-based funding models attempt to reward institutions 
for meeting workforce needs. As described above, states typically do this by placing more 
weight upon graduates in fields deemed as high-priority by the state. As described above, meeting 
workforce needs can be considered a process metric designed to improve labor-market outcomes 
for students and to ensure the institution is responsive to state and local needs. However, current 
approaches to identifying workforce need leave a lot to be desired, so it is unclear whether current 
efforts will achieve their well-intentioned effects. 


Past and current outcomes-based funding models often do include access and diversity 
metrics. As described above, outcomes-based funding models often place greater weight upon 
degrees awarded to at-risk students. This is a direct way to reward institutions for promoting access 


and diversity. 


A handful of states have successfully 
implemented outcomes-based funding A handful of states have successtully 
models that include more explicit quality 


implemented outcomes-based funding 
metrics. Missouri's outcomes-based funding 


model includes performance on general and models that include more explicit 
major-specific assessments (National Governors quality metrics. 

Association, 2013). Nevada and Tennessee nnunnnnnnnnnnnnne 
both include student job placement as a metric 

within their outcomes-based funding model (National Governors Association, 2013). The Texas 
State Technical College System recently shifted to a funding model that is based entirely on the 


earnings of its graduates. 
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Over time, as a research base 


While past and present outcomes-based funding emerges and effective processes 
models have implicitly or explicitly addressed quality 
in the ways described above, they can and should go are identified, outcomes-based 
further to ensure that efforts to improve productivity funding models can include 
enhance rather than undermine quality. Specifically, . 2 ils 
states should consider including value-added metrics of weight UPON Process indicators 
student labor-market outcomes, as well as measures of — that are linked directly to desired 
student learning and engagement. More work should auicenes 
be done to help states identify statewide and local 
fork forme needsse thavelicchine proeceeanciicecia. Penner tperetreri mene 
be designed to reward institutions for addressing those needs. States can use student learning met- 
rics to monitor quality as they implement outcomes-based funding models. Finally, research should 
identify additional processes that improve student labor-market outcomes, learning and engagement. 
Over time, as a research base emerges and effective processes are identified, outcomes-based funding 


models can include weight upon process indicators that are linked directly to desired outcomes. 


Conclusion and Recommendations 


To better align institutional priorities and activities with state goals and to more efficiently pri- 
oritize dwindling state resources for higher education, states are increasingly seeking to implement 
outcomes-based funding models that explicitly tie institution funding to a set of performance met- 
rics. Policymakers need sound information to make these decisions. While current research on these 
issues is only just emerging, several key themes are evident. 


States should consider ways to draw upon and shape the emerging research agenda on 
measuring quality in higher education so as to ensure that outcomes-based funding models 
enhance quality while also improving studemt (——.v:seesessseeseeeccescceeceeesesseesseeeceeecseceseeseseseneeeeeees 


progression and completion. Existing quality Po| icymakers and researchers 
metrics vary considerably in the level and quality 


of research and testing that has supported their should work together fo guide 


development and refinement. Given the current state the development, implementation, 

of research and implementation, states must develop a sexi f nee ini 

a pragmatic approach to incorporating emerging nO SV ON OO OO Se OF Tens 

quality metrics into funding models. Policymakers that accurately measure the 
d hers shoul k her to guide th ; 

and researc ers shou d worl together to guide t e Banus Marsares would like 

development, implementation, and evaluation of a 

set of metrics that accurately measure the constructs !O0 Encourage. 

that states would like to encourage. Given the StAtC  —srwnnvccccccccncccccssccccsccecscccensccssssceesccssessesnsccesssees 


of research and implementation, we recommend that: 


States should consider using student grades and learning metrics to monitor quality as 
they implement outcomes-based funding models. Critics of outcomes-based funding argue 
that increased productivity will be accompanied by watered-down curricula and reduced student 
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While it is challenging to obtain buy-in 
learning. While it is challenging to obtain tg include student assessments directly 
buy-in to include student assessments 


directly into outcomes-based funding into outcomes-based tunding models, 


models, a program of low-stakes assessments q program of low-stakes assessments 
that are used to monitor quality as outcomes- jos ae 7 li 
based funding is implemented may be more ae Sed ION ity oe 


acceptable. If student learning declines as outcomes-based tunding iS implemented 


tes impl t - fundi 
states implement outcomes-based funding, may he more acceptable. Pcudent 
then further steps can be taken to ensure 
quality. As an additional check, states may lea ming declines as states implement 
consider monitoring student grades for evi- outcomes-based funding, then fy rther 
dence of grade inflation as they implement 


outcomes-based funding models. If grades steps can be taken to ensure quality. 
increase without ACONCOMItAaNt INCLEA|SE Ir “TTT Tee eect ee cece eeeee eee eeereeeseeseeeeeesues 
student assessment scores, then this may be taken as a sign of increased grade inflation that would 
trigger increased focus on quality. 


States should begin to incorporate appropriately researched quality metrics into 
funding models. 


The most promising college quality metrics for current use in outcomes-based funding models 

are input-adjusted measures of student outcomes and student access and diversity. These metrics are straight- 
forward and understandable, draw from existing data sources, and are acceptable to stakeholders. 
Adjusting student outcomes for student characteristics can help make fairer comparisons across 
colleges and universities with different types of students. 


Licensure tests are attractive sources of data for measuring student learning at the program level. 
These tests provide a uniform measure of the knowledge and skills of graduates of the programs 
that require licensure tests that all students must achieve to enter the profession. By adjusting 
student-level scores for entrance-exam scores, states can develop value-added measures of student 
learning in particular programs. 


Undertake a program of research to develop, implement and evaluate appropriate methods 
for identifying workforce needs. Meeting workforce needs is a laudable goal of colleges and 
universities and one for which state governments would like to create incentives. However, current 
methods are limited in their ability to pinpoint workforce needs. Further research is needed to help 
states develop appropriate metrics for rewarding colleges and universities that meet these needs. 


Aim towards including process metrics. Process metrics are attractive, but more research is 
needed to identify processes that produce desired outcomes. Process metrics have proven useful in 
other government sectors like healthcare and K-12 education. Institutions may want to improve 
performance along a set of output or outcome metrics, but may not know what processes can 

help them to do so. By directly including process metrics in the outcomes-based funding model 

or developing additional incentive structures that directly fund such processes, the state can help 
institutions implement effective processes that are aligned with state goals. The research base on 
the impact of different institution processes on relevant outcomes is quite limited, but is rapidly 
emerging. 
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Wherever possible, student outcomes and diversity metrics for outcomes-based funding 
models should draw upon existing data. State administrative data are a powerful and cost effective 
resource for developing student outcomes and diversity metrics. These data are available in most 
states and allow researchers to adjust student outcomes for student characteristics. Attractive federal 
data sources include the Integrated Postsecondary Education Data System (IPEDS) and the National 
Postsecondary Student Aid Survey (NPSAS). 


Balance the need to adjust student outcomes for student characteristics with the need to 
be straightforward and understandable. Student outcomes metrics should account for colleges 
and universities serving different populations by adjusting outcomes for student characteristics. 
Regression-based methodologies that are popular with researchers can be overly complex for a 
funding formula, making outcomes less transparent and difficult to communicate to a broad set of 
stakeholders. A simpler metric based on conditional means for outcomes of students with particular 
characteristics can address the need to adjust for student characteristics while also remaining under- 
standable. Another simpler approach that has seen wider implementation is to place greater weight 
upon outcomes achieved for at-risk students. 


Stakeholder buy-in is key. Experience with 
outcomes-based funding models has shown that Stakeholders should agres on a sei of 


it is imperative to involve stakeholders in the metrics that represent their collective 


model development process. Stakeholders ; : 
should agree on a set of metrics that represent goals for higher education and a 


their collective goals for higher education method for tying those metrics to state 
and a method for tying those metrics to state 


funding. To support the engagement process, funding. To support the engagement 


states should highlight ways in which current process, states should highlight ways 
outcomes-based funding models directly or : biel Ls d 
oe tins ; in which current ouftcomes-base 
indirectly provide incentives for quality. Future 

approaches should more explicitly incorporate funding models directly or indirectly 


direct quality metrics in the funding model, and provide incentives for quality. 


employ student learning metrics to monitor 
quality as new funding models are implemented. 
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